KVM local migration issue #3521 by GabrielBrascher · Pull Request #3533 · apache/cloudstack

GabrielBrascher · 2019-07-31T07:49:12Z

Description

Fix regression bug that affects KVM local storage migration. Some of the desired execution flows for KVM local storage migration had been altered to allow only managed storage to execute. Fixed allowing managed and non managed storages to execute.

Fixes #3521

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

Start a VM on a KVM host with local storage;
Migrate the VM to another KVM host that has local storage
Assert that (i) ACS volumes table, (ii) the VM volume content, and (iii) the VM libvirt xml have all been updated correctly.
Keep migrating (repeating steps 2 and 3) the VM around hosts (also migrate back the VM to its first host)

Local storage KVM live migration does not support VIR_MIGRATE_NON_SHARED_INC. Among other fixes that address apache#3521, this commit allows to use flag VIR_MIGRATE_NON_SHARED_DISK when migrating local storage VMs.

yadvr · 2019-07-31T08:05:45Z

@blueorangutan package

blueorangutan · 2019-07-31T08:06:45Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-07-31T08:27:42Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-202

yadvr · 2019-07-31T10:10:06Z

Live storage migration for vmware and kvm for VMs with shared storage would be needed cc @andrijapanicsb @borisstoyanov @nvazquez
@blueorangutan test matrix

blueorangutan · 2019-07-31T10:10:46Z

@rhtyd a Trillian-Jenkins matrix job (centos6 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2019-07-31T20:01:44Z

Trillian test result (tid-249)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33429 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3533-t249-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2019-07-31T20:26:41Z

Trillian test result (tid-248)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34904 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3533-t248-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Smoke tests completed. 75 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_list_snapshots	`Error`	0.05	test_list_ids_parameter.py
test_01_scale_vm	`Failure`	32.01	test_scale_vm.py

blueorangutan · 2019-08-01T00:29:51Z

Trillian test result (tid-250)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49577 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3533-t250-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vgpu_enabled_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 75 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_snapshots_with_removed_data_store	`Error`	1.22	test_snapshots.py
test_01_volume_usage	`Error`	10.47	test_usage.py

yadvr · 2019-08-02T15:15:45Z

Tests LGTM, can you review @andrijapanicsb @borisstoyanov - we'll need to test live vm with storage migration of shared/nfs storage on KVM

yadvr · 2019-08-02T15:17:30Z

+
+            StoragePoolVO destStoragePool = _storagePoolDao.findById(destDataStore.getId());
+            StoragePoolVO sourceStoragePool = _storagePoolDao.findById(srcVolumeInfo.getPoolId());
+            if (sourceStoragePool.getPoolType() == StoragePoolType.NetworkFilesystem && destStoragePool.getPoolType() == StoragePoolType.NetworkFilesystem) {


@GabrielBrascher shouldn't it loop and check for all volumes before returning true? It could be like it can return false if any of the src/dest pools are not nfs

@rhtyd The idea is that if at least one of the disks is NFS it cannot be migrated on the execution flow of local storage. Taking a second look I see that the method name might need enhancements.

Am I missing something? Please let me know if something doesn't look right. Thanks for the review.

yadvr

One remark left, otherwise LGTM

andrijapanicsb · 2019-08-06T11:29:41Z

CentOS6, VM+ROOT on local storage, fails in CloudStack with following lines on the destination KVM host:

2019-08-06 12:08:05,608 INFO [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:a900bfb2) Trying to fetch storage pool 7d69d5bf-0383-476a-90bd-41139db0d596 from libvirt
2019-08-06 12:08:05,610 WARN [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:a900bfb2) Caught:
com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 7d69d5bf-0383-476a-90bd-41139db0d596 from libvirt
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:256)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:242)

The storage pool 7d69d5bf-0383-476a-90bd-41139db0d596 is the local pool on SOURCE host.

@GabrielBrascher not sure if you want this fixed on CentOS6, since we officially support it ?

Ubuntu 18, VM+ROOT on local storage - migration to another host works fine.

CentOS7 doesn't work (thanks to RedHat $$$hitty commercial logic and qemu code changes...)

Will test shared storage volume migration...

andrijapanicsb · 2019-08-06T11:31:01Z

@mike-tutkowski any chance to test live storage migration with KVM+SolidFire - I see a lot of removed/changed code "xxx_managed_yyy"?

@GabrielBrascher can you advise on above ^^ ^?

andrijapanicsb · 2019-08-06T11:38:40Z

@syed ^^^

GabrielBrascher · 2019-08-06T13:04:28Z

Thanks for testing @andrijapanicsb. I did all tests on KVM + Ubuntu and it worked well; however, I did not test with CentOS6. Is there any log for share so that we can tackle the CentOS6 issue?

andrijapanicsb · 2019-08-06T16:16:15Z

@svenvogel can you kindly test if KVM live migration with SF works fine (NFS/CEPH to SF) ?

andrijapanicsb · 2019-08-06T16:28:24Z

@GabrielBrascher
here they are:
mgmt logs: https://pastebin.com/L97FgFut
agent logs: https://pastebin.com/D6HnZSnJ

andrijapanicsb

LGTM except previously left 2 comments @GabrielBrascher @rhtyd

centos6 support for locale storage migartion - yes or no ?
would be good that someone test CEPH/NFS to SF if possible with KVM (won't work on CentOS 7 except if very custom qemu/libvirt used)

=====================================
For the record, what has been tested for regression

Baseline was 4.11.3

4.11 KVM:
NFS to NFS - online storage migration (migrateVirtualMachineWithVolume) doesn't work at all (1 or 2 volumes attached to VM). This might have been fixed in 4.12
NFS to NFS - offline migration of VM works only when single ROOT disk attached (fails if DATA disks attached), otherwise direct volume offline migration works fine.

4.11 VMware -
NFS to NFS - online storage migration (migrateVirtualMachineWithVolumes) - works fine for both single disk VM and VM with DATA disks as well
NFS to NFS - offline migration of VM works only when single ROOT disk attached (fails if DATA disks attached), otherwise direct volume offline migration works fine.

==============================
4.13 PR3533 - KVM:
NFS to NFS - online storage migration (migrateVirtualMachineWithVolumes) with just ROOT volume attached works (fails if DATA disks attached, even if migrating choosen to migrate just the ROOT volume)
NFS to NFS - offline migration ov VM works same as in 4.11 (can migrate VM with only ROOT volume, otherwise direct volume offline migration works fine)

4.13 PR3533 VMware
NFS to NFS - online storage migration (migrateVirtualMachineWithVolumes) - works fine for both single disk VM and VM with DATA disks as well
NFS to NFS - offline migration of VM - an improvement over 4.11 - can migrate the whole VM with both ROOT and DATA volumes attached

borisstoyanov

LGTM based on test results and Andrija's testing

LGTM

yadvr · 2019-08-07T10:11:24Z

I think for EL6, storage migration (live) would be limiting. The feature should largely be encouraged on newer distros with newer libvirt.

andrijapanicsb · 2019-10-31T12:36:47Z

@GabrielBrascher need your advice since you guys worked on the local migration with KVM.

I saw a case (didn't reproduce it myself, but while working with the customer, on 4.13) that when KVM is using local storage, and you try to put a host into maintenance, the following seems to happen:

User VMs - not sure if they are migrated away - can't say for sure - but I assume they are.
CPVM and SSVM are NOT live migrated (with storage), fails as before
VRs are live migrated with their storage/volume to a new host - BUT the volume format in DB seems to be set to RAW (again, this is based other-users data, not my own testing - from the cloud.volumes table)

My question is - what is expected to be happening and what is not, and why is the VR's volume in RAW, vs QCOW2 format? Can you lay down the expected behaviour based on the code you've done so far (in 4.13)

Thanks

GabrielBrascher · 2019-11-06T12:44:30Z

@andrijapanicsb I will run some tests and check this out, thanks for the heads-up.

GabrielBrascher added 2 commits July 25, 2019 20:05

Fix regression bug that affects KVM local storage migration

dd9d530

Allow migrating Non Shared Disk for non managed storage.

ac2309c

Local storage KVM live migration does not support VIR_MIGRATE_NON_SHARED_INC. Among other fixes that address apache#3521, this commit allows to use flag VIR_MIGRATE_NON_SHARED_DISK when migrating local storage VMs.

GabrielBrascher added component:kvm Severity:BLOCKER component:local-storage labels Jul 31, 2019

GabrielBrascher added this to the 4.13.0.0 milestone Jul 31, 2019

GabrielBrascher self-assigned this Jul 31, 2019

yadvr requested a review from nvazquez July 31, 2019 08:07

yadvr requested review from andrijapanicsb, borisstoyanov and nvazquez and removed request for nvazquez July 31, 2019 10:09

yadvr reviewed Aug 2, 2019

View reviewed changes

yadvr previously requested changes Aug 2, 2019

View reviewed changes

andrijapanicsb approved these changes Aug 6, 2019

View reviewed changes

borisstoyanov approved these changes Aug 7, 2019

View reviewed changes

yadvr merged commit 5dc982d into apache:master Aug 7, 2019

Conversation

GabrielBrascher commented Jul 31, 2019

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

yadvr commented Jul 31, 2019

Uh oh!

blueorangutan commented Jul 31, 2019

Uh oh!

blueorangutan commented Jul 31, 2019

Uh oh!

yadvr commented Jul 31, 2019

Uh oh!

blueorangutan commented Jul 31, 2019

Uh oh!

blueorangutan commented Jul 31, 2019

Uh oh!

blueorangutan commented Jul 31, 2019

Uh oh!

blueorangutan commented Aug 1, 2019

Uh oh!

yadvr commented Aug 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr Aug 2, 2019

Choose a reason for hiding this comment

Uh oh!

GabrielBrascher Aug 6, 2019

Choose a reason for hiding this comment

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

andrijapanicsb commented Aug 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrijapanicsb commented Aug 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrijapanicsb commented Aug 6, 2019

Uh oh!

GabrielBrascher commented Aug 6, 2019

Uh oh!

andrijapanicsb commented Aug 6, 2019

Uh oh!

andrijapanicsb commented Aug 6, 2019

Uh oh!

andrijapanicsb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Aug 7, 2019

Uh oh!

andrijapanicsb commented Oct 31, 2019

Uh oh!

GabrielBrascher commented Nov 6, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yadvr commented Aug 2, 2019 •

edited

Loading

andrijapanicsb commented Aug 6, 2019 •

edited

Loading

andrijapanicsb commented Aug 6, 2019 •

edited

Loading

andrijapanicsb left a comment •

edited

Loading